There is a nonvolatile semiconductor memory device having a three-dimensional structure. For example, a through-hole is made in a stacked body in which electrode films and insulating films are multiply stacked alternately; and a semiconductor pillar that is used to form a channel is provided on the side wall of the through-hole with a charge storage layer interposed. It is desirable to further suppress the fluctuation of the characteristics of such a nonvolatile semiconductor memory device.